Photography is the process of making pictures by means of the action of light. Light is the commonly used term for electromagnetic radiation in a frequency range that is visible to the human eye. Light patterns reflected or emitted from objects are recorded by an image sensor through a timed exposure. Image sensors can be chemical in nature, such as photographic film, or solid state in nature, such as the CCD and CMOS image sensors employed by digital still and video cameras.
Digital cameras have a series of lenses that focus light to create an image of a scene. But instead of focusing this light onto a piece of film, as in traditional cameras, it focuses it onto the image sensor which converts the electromagnetic radiation of the light into an electrical charge. The image sensor is said to be a picture element, or a ‘pixel.’ The electrical charge indicates a relative intensity of the electromagnetic radiation as perceived by the image sensor, and generally is used to associate a light intensity value with the pixel.
FIG. 1 illustrates several components that may be included in one possible implementation 10 by which a natural scene is captured to form an electronic image. System 10 includes a signal source 100 and a signal processing chain that includes integrator 110, analog to digital converter (ADC) 120 and digital signal processor (DSP) 130.
The output of integrator 110, VOUT, is input to ADC 120. ADC 120 performs the analog to digital conversion function. The analog to digital conversion function is well known in the art. The analog signal VOUT that is present at the input of ADC 120 is converted into signal VD that is present at the output of ADC 120. VD can assume one of a set of discrete levels usually but not always measured in units of volts. By way of example another unit of measure for the output of ADC 120 can be amperes.
By way of example signal source 100 could be a light intensity sensor that is used in a timed application, such as in a digital camera application where the sensor is exposed to the incoming light for a specific duration of time that is commonly referred to as the exposure time. The integrator 110 then serves the function of integrating the responses of sensor 100 caused by all photons received during the exposure time into one output value to be read out at the end of the exposure time. By way of example the integrator 110 output value could be a voltage measured in units of volts.
FIG. 2 is a simplified illustration of a potential image sensor structure block diagram. Signal source 1000 is a light sensor that by way of example could be a photodiode. Component 1040 is a simple integrator that by way of example could be a capacitor. The input to the integrator is the output of signal source 1000. Integrator 1040 is reset by switch 1050 which is in the closed position prior to starting the integration process. The ability to rapidly reset the state of integrator 1040 is an important aspect of the image sensor operation. At the start of the integration process switch 1050 opens and the voltage across integrator 1040 begins to change in response to the input signal originating from signal source 1000. At the end of the integration process switch 1030 closes and integrator output 1060, VOUT, is sampled. FIG. 2 is an illustrative diagram and the implementation of other similar image sensor structures with identical functionality is well known to one skilled in the art.
In an alternative and equivalent mode of operation of the simplified image sensor structure block diagram of FIG. 2 the integrator embodied by way of example by capacitor 1040 is reset by switch 1050 to a high voltage V+ or POWER instead of to V− or GROUND. At the start of the integration process switch 1050 opens and the voltage across integrator 1040 begins to change in response to the input signal originating from signal source 1000. At the end of the integration process switch 1030 closes and integrator output 1060, VOUT, is sampled.
The simplified block diagram of an image sensor structure illustrated in FIG. 2 by way of example is subject to some performance limitations. One such limitation, the dynamic range, is described here by way of example together with an explanation that gives insight into its causes. The integrator output 1060, VOUT, cannot in general exceed the upper limit imposed by the available power supply voltage. Power supply voltages are decreasing in state-of-the-art equipment due to stringent power consumption requirements. Integrator output 1060 cannot exceed the power supply voltage and will saturate if the integrator output signal attempts to build up after reaching the power supply voltage level.
Saturation occurs when the output voltage reaches the available power supply voltage and is unable to respond to further changes in the input signal. Signal saturation causes system performance degradation. FIGS. 3A through 3C illustrate potential distortions at the output of a pixel structure consisting of light sensor 100 and integrator 110 due to the dynamic range limitation of the photosensitive element structure and more specifically of the integrator structure. They also illustrate potential distortions at the output of light sensor 1000 and integrator output 1060 due to the dynamic range limitations. Segment (a) of FIG. 3(a) illustrates the linear increase of integrator 110 output in response to a constant input signal of different level. It also illustrates the linear increase of integrator output 1060 in response to a constant input signal of different level. The image sensor structure will perform well for the range of input light intensities that give rise to the linear output of segment (a); the image sensor structure will not perform well for the range of input light intensities that give rise to the saturated output of segment (b).
The integrator output response is indicative of limited dynamic range. As illustrated in FIG. 3(a) one version of the embodiment of the image sensor of FIG. 2 will render well shadow detail but will fail to render highlight detail. It is possible to shift the response as illustrated in FIGS. 3(b) and 3(c). In FIGS. 3(b) and 3(c) the dynamic range of the image sensor remains the same but the response characteristic is shifted. The response characteristic of FIG. 3(b) loses shadow and highlight detail but retains good midrange response. The response characteristic of FIG. 3(c) looses shadow detail and partial midrange detail in order to maintain good highlight detail.
FIG. 4A illustrates the histogram of the pixel intensities of an overexposed image capture where a multitude of image sensors that were exposed to the image were driven into saturation. As seen in FIG. 4A the maximum image sensor structure output value is ‘255’ and the units used are the ADC 120 output corresponding to the image sensor output voltage. The light intensity caused many light sensors 100 to output a value that saturated the integrator 110 as the exposure progressed during the exposure period. The maximum (saturated) value of the output of integrator 110 caused the ADC to generate the output code ‘255’ which is the maximum output code for an 8-bit ADC. The image capture will be of lower than optimal quality due to the inability of those image sensors subject to high intensity light inputs to achieve a sufficiently high output level.
The distortion illustrated in the histogram of FIG. 4A corresponds to the individual pixel distortion. A lower exposure time would have caused the outputs of the image sensors subject to high intensity light inputs to register an output level below 255 and avoid the high end distortion but would have prevented the image sensors subject to low intensity light inputs to remain at an output level value of 0 and not register the light intensity details contained in the shadows and other low light image segments.
FIG. 4B illustrates the histogram of the pixel intensities of an underexposed image capture where a multitude of image sensors were not exposed to sufficient light to achieve a minimum output value above ‘0’. As seen in FIG. 14B the minimum image sensor structure output value is ‘0’. The units refer to the ADC 120 output levels corresponding to individual image sensor structures. The light intensity received at the image sensor caused many individual light sensors 100 to output a value that failed to cause integrator 110 to output a sufficiently high value to cause a minimal ADC output code above ‘0’ as the exposure progressed during the exposure period. The image capture will be of lower than optimal quality due to the inability of those image sensors subject to low intensity light inputs to generate a sufficiently high response level. The distortion illustrated in the histogram of FIG. 4B corresponds to the individual pixel distortion. A longer exposure would have caused the outputs of the individual image sensors subject to low intensity light inputs to register an above ‘0’ output and avoid the low end distortion but would have also caused the image sensors structures subject to high intensity light inputs to saturate at a ‘255’ output value and not register the light intensity details contained in the highlights and other bright light image segments.
FIG. 5A illustrates the response of yet another of the two or more solid-state image sensor structures used to build the heterogeneous image sensor disclosed in this patent application. The extended dynamic range of the solid-state image sensor structure is sufficient to produce an image sensor response over the full range of electromagnetic radiation intensity impinging upon the image sensor structure. This enables the solid-state image sensor structure to capture sufficient charges in the darkest portion while avoiding the saturation affects in the brightness portions of the image to be captured. The net effect it is faithful reproduction of the image to be captured regardless of whether the light from the darkest segment or the light from the brightest segment of the scene to be captured is impinging upon the image sensor. FIG. 5B illustrates the histogram of the pixel intensities of a correctly exposed image capture where all image sensor outputs are within the dynamic range of the 8-bit ADC that is ‘0’ to ‘255’.
Integrator saturation before the end of the exposure period is a limiting factor in the dynamic range of an image sensor structure. Solutions to the integrator saturation problem have been published. The feature the published solutions have in common is the monitoring of the integrator output to detect the onset of saturation condition at which time the integrator is discharged and the event is recorded.
By way of example of such solutions Mazzucco discloses in U.S. Pat. No. 6,407,610 methods to prevent saturation of the integrator output. The prevention methods consist of sensing the onset of saturation and resetting (discharging) the integrator or changing the direction of integration when the onset of saturation is sensed. An external circuit records all such events. At the end of the integration period the effective full range of the integration is reconstructed from the number of recorded reset events and from the final integrator output voltage. A similar approach is disclosed by Merill in U.S. Pat. No. 6,130,713.
All such solutions have in common the need to compensate for the dynamic range limitation inherent to the native image sensor structure. These solutions require the introduction of additional circuit elements into the image sensor structure in order to perform the functions outlined in the disclosures.
These solutions have in common the independent operation of each image sensor with respect to other image sensors and the need to incorporate the additional circuit elements into each image sensor capable of extended dynamic range performance. The introduction of the additional circuit elements causes complexity, cost and power consumption to increase while the device yields and reliability decrease.